Skip to content

fix(analyzer): keep wrapped inline code spans across inline HTML - #1085

Open
Koloyaaa wants to merge 1 commit into
NVIDIA:mainfrom
Koloyaaa:fix/1023-wrapped-inline-code-span
Open

Koloyaaa wants to merge 1 commit into
NVIDIA:mainfrom
Koloyaaa:fix/1023-wrapped-inline-code-span

Conversation

@Koloyaaa

Copy link
Copy Markdown

Fixes #1023.

Impact: silent-wrong-result

Who reaches this: any skillspector scan of a Markdown file with an inline code span that wraps across a soft line break, where the continuation line starts with an inline-HTML-looking tag such as <path>.

Root cause: _markdown_shell_text treats any line that begins with < plus a tag-name prefix as an HTML block start. In the issue's fixture, line 2 begins with <path>, so the projection treats the rest of the paragraph as an HTML body, leaves every backtick visible, and the shell tokenizer reads the closing delimiter at offset 177 as an unmatched substitution opener. The scan reports static_parse_limit: references/guide.md is 50% covered and the linking SKILL.md gets AE1 HIGH (score 25, CAUTION).

Fix: a <... continuation line may interrupt an open paragraph only when it is a real CommonMark HTML block start — a raw-text tag, comment, processing instruction, declaration, CDATA section, or one of the type-6 block tags. Otherwise the line stays in the paragraph and backtick runs pair across the soft break. The type-6 tag list mirrors the repository's security_reconstruction._HTML_BLOCK_TAG_RE.

Tests: tests/nodes/analyzers/test_static_parse_limit_issue_1023.py covers projection masking, bounded-parse completion, ledger COMPLETED, the joined control, preserved <pre> and <div> blocks, and an end-to-end CLI scan of the issue fixture. Before the fix the fixture scans at 50% coverage with AE1 HIGH; after the fix it scans at 100%, complete, score 0, SAFE, including with --fail-on-incomplete.

Verification: focused file 7 passed; planned Markdown/parser subset 1,355 passed, 1 deselected (POSIX-only file-output test); security-reconstruction, false-positive, and parse-limit group 1,036 passed; ruff check and ruff format --check clean. make test is not available on the Windows host used here because the integration target may call an LLM provider; CI will run the full suite.

Known limitation: the guard requires a pending inline-code run, so the no-pending-run <path> shape remains an HTML block. The second double-backtick case from the issue is not part of this change.

Signed-off-by: Koloyaaa yakamozzz@sjtu.edu.cn

@yashrajp22 yashrajp22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wrapped inline-tag continuation is handled, but a new list item can inherit the preceding paragraph's inline-code state and mask literal HTML backticks. Please preserve that block boundary.

html_open = re.match(r"<(?:[A-Za-z][A-Za-z0-9-]*(?=[\s/>]|$)|[!?/])", leading)
if (
html_open
and paragraph_open

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please keep HTML-block detection active when this line starts a new list item. For example:

Intro `open
- <custom>
  `rm -rf /`

Here paragraph_open and runs still belong to Intro, so the guard clears html_open before the new list marker closes that paragraph. <custom> starts a real type-7 HTML block inside the new item, whose body backticks must remain literal. Instead, they are masked and registered as inline-code ranges, changing the shell parser's input. Please limit this guard to continuation of the same paragraph, excluding a new list item.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Decision: Changes Requested

Value and readiness

The wrapped-code-span fix addresses a real completeness false positive in ordinary Markdown, and the added tests cover the issue fixture through projection, ledger and CLI output. One block-boundary regression still needs correction before this is ready.

Material findings

Preserve new-list HTML block ownership. In src/skillspector/nodes/analyzers/static_patterns_tool_misuse.py:9068, the new guard uses paragraph_open and runs from the preceding paragraph after the current list marker has already been stripped. For the existing reviewer example:

Intro `open
- <custom>
  `rm -rf /`

has_list_marker is true, but the guard clears html_open. The later new-item branch clears the old runs without entering the <custom> HTML block, so the following raw HTML-body backticks are processed and masked as inline-code delimiters. This changes the security parser's input and grants ownership that the rendered block does not establish. Restrict this exception to continuation of the same paragraph, excluding a new list item, and add a regression asserting the HTML-body ticks remain literal. This confirms the still-open current-head thread; there is no author fix or reply in the frozen inventory.

PIC tradeoffs

No new product decision is needed. The documented second issue variant can remain outside this patch; preserving actual Markdown block boundaries is required for the chosen conservative projection.

Verification and gaps

Reviewed the complete two-file diff, exact-head surrounding paragraph/list/HTML state transitions, all seven added tests, linked issue #1023, the complete available review/comment/thread records and all six successful hosted check results. Pre-fetch, post-fetch and post-review heads match the frozen inventory. The review thread remains unresolved and the other review requests changes. No contributor code, tests, imports, builds, hooks, scanner, regex probes or model calls were executed.

Reviewed head: fbc54d17bfb35bc19b06ba065ffe8b5e720af043.

Koloyaaa added a commit to Koloyaaa/SkillSpector that referenced this pull request Nov 13, 2026
A list marker ends the preceding paragraph, so its content starts a
fresh block where an inline tag can open a real HTML block. The
wrapped-span guard now applies only to same-paragraph continuation,
as requested in review on NVIDIA#1085.

Signed-off-by: Koloyaaa <yakamozzz@sjtu.edu.cn>
@Koloyaaa

Copy link
Copy Markdown
Author

Updated in 6a21283: the guard now also requires the line to continue the same paragraph (and not has_list_marker), so a new list item keeps HTML-block detection and its HTML-body backticks stay literal. Added a regression test using the example from review (Intro with an open backtick, then a - <custom> item, then the indented rm -rf line), asserting the projection is byte-identical.

Re-verified: focused file 8 passed; planned Markdown/parser subset 1,356 passed / 1 deselected; security/parse-limit group 1,036 passed; ruff check and format clean. The #1023 fixture still scans complete at 100% coverage, score 0.

@yashrajp22 yashrajp22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new-list HTML boundary is addressed. One remaining case lets a literal HTML-attribute backtick create a false inline-code range and combine separate lines into an unsafe-option match.

if (
html_open
and paragraph_open
and runs

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please verify that runs contains a real open code span before using it to clear html_open. For example:

Intro <a title="`">curl
<custom> --insecure
`done`

The first backtick belongs to the quoted HTML attribute, but it remains in runs. This guard then clears html_open on line 2, and paragraph pairing matches that attribute tick with the opener before done, leaving the real closing tick unmatched. The resulting inline_regions also makes _option_command_matches join curl with --insecure across the intervening markup. Please exclude HTML-owned literal ticks from the pending-span proof, or keep the conservative behavior when ownership is uncertain.

@Koloyaaa

Copy link
Copy Markdown
Author

Pushed a follow-up revision. Backticks inside complete inline HTML tags no longer count as evidence of a pending code span, and a tag whose attributes wrap within the same paragraph still owns its ticks.

Your example and its wrapped variant now pass through unchanged, with no inline region and no curl/--insecure join:

Intro <a title="`">curl
<custom> --insecure
`done`

An unclosed tag still leaves its backticks as code delimiters; I checked that against markdown-it-py.

11 focused tests pass, plus the 1,348-file Markdown/parser subset, the 1,036-test security group, and ruff. I left the review thread unresolved so you can verify.

@yashrajp22 yashrajp22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The list boundary and original attribute example are handled. The earlier attribute issue remains for attributes on later lines; the inline comments below cover two additional problems with code-span precedence and repeated paragraph scans.

tag whose attributes wrap within a paragraph still owns its ticks.
"""
limit = len(text) if end is None else end
tags = _markdown_html_tag_ranges(text, start, limit if tag_limit is None else tag_limit)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please preserve code-span ownership before excluding HTML-looking ranges. For example:

Run `curl
--insecure <name`>.

The code span owns both backticks, but this pass treats <name through > as a tag and discards the closing tick. No inline region is produced, so _option_command_matches treats the newline as a command boundary and misses curl --insecure. The same ordering problem can also extend a span past its real closer into later prose. Please recognize inline constructs in source order and require valid, unescaped HTML before granting it ownership.

for match in backtick_runs.finditer(line):
if paragraph_tag_limit is None:
paragraph_tag_limit = _markdown_paragraph_limit(content, offset)
for start, end in _markdown_backtick_runs_outside_html(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please avoid rescanning the remaining paragraph for every physical line. paragraph_tag_limit stays at the paragraph end, and _markdown_backtick_runs_outside_html builds all tag ranges up to that limit before looking for any backticks. A paragraph of 10,000 a <b> lines therefore visits 50,005,000 tags, even with no backticks, turning this previously linear projection into quadratic work. This consumes the scanner's shared runtime budget and can leave ordinary Markdown incompletely inspected. Please compute ownership once per paragraph or carry a forward parser state, with runtime checks inside long scans.

@Koloyaaa

Copy link
Copy Markdown
Author

Pushed a revision that covers both points.

Ownership now happens in source order. A code span that opens before a tag keeps its body, so the closing tick in this example is honored again:

Run `curl
--insecure <name`>.

It produces the region curl\n--insecure <name and the curl/--insecure match; the fixture now reports TM1 instead of passing without a finding.

Tags need to match CommonMark's inline HTML grammar and be unescaped before they can own a tick. An unclosed or malformed < stays literal.

The per-line paragraph scan is gone. Tag ranges are computed once per paragraph, and a 40,000-line paragraph of a <b> line takes about 0.18s here; a call-count test pins one computation per paragraph.

14 focused tests pass, along with the 1,348-file Markdown/parser subset, the 1,036-test security group, and ruff. I left the review thread open so you can check it.

@Koloyaaa

Copy link
Copy Markdown
Author

CI found five failing cases in the new dense-reference test, all from the same paragraph scan, so I pushed a follow-up.

That test separates four-line blocks with headings and no blank lines. The scan used to stop only at blank lines, so it walked to the end of the file for each paragraph. At 256 KB that became quadratic and the ledger reported PARTIAL. Paragraphs now also end at headings, blockquotes, list items, fences, thematic breaks, and HTML blocks, which is where CommonMark lets a block interrupt a paragraph.

The 256 KB case now scans 220,682 of 262,144 characters in about 0.28s, and the dense test passes 15/15 here. The focused tests, the Markdown/parser subset, and the security group still pass, and the earlier code-span cases are unchanged.

I left the review thread open.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Decision — Request changes.

Value and readiness — The current revision preserves the wrapped placeholder span and addresses the prior list-boundary, HTML-attribute, and code-span precedence examples. Computing the HTML ranges once removes the earlier per-line parse, but the subsequent filtering still repeats work.

Material findings

  • static_patterns_tool_misuse.py:8965–8968 restarts tag_index at zero for every physical line in a paragraph. In a long paragraph with one inline tag and one complete inline code span per line, each line with backticks walks all tag ranges from earlier lines. The visits grow quadratically, and the inner loop contains no runtime check. The new call-count test uses lines without backticks, so that scan never runs there. Carry a forward range cursor through the paragraph or bisect sorted tag endpoints at the line's start. Add a dense tag-and-backtick control that measures this filtering work and checks that the projection and ledger remain complete. Tag filtering · Per-line caller

PIC tradeoffs — Retaining source-order ownership is valuable for accurate findings. Indexing the already computed ranges keeps the precision gain while guarding the analyzer's coverage budget on reference-heavy Markdown.

Verification and gaps — Reviewed the complete diff, paragraph and list state, ownership/filtering paths, fixtures, and prior threads and replies. All six recorded checks succeeded for this head, and GitHub reports a merge conflict. Static review only.

Reviewed head: c3554b53cd361bf8b3767d45bf35dc9faba0cd31

Current main already pairs the wrapped span correctly, so this adds
the issue fixture and its joined control as a regression test. The
test fails on bb991c5 and passes on current main.

Signed-off-by: Koloyaaa <yakamozzz@sjtu.edu.cn>
@Koloyaaa
Koloyaaa force-pushed the fix/1023-wrapped-inline-code-span branch from c3554b5 to 3ca711b Compare November 20, 2026 01:12
@Koloyaaa

Copy link
Copy Markdown
Author

I reduced this PR to a regression test.

Current main already handles the reported wrapped span: the issue fixture scans at 100% coverage, complete, score 0, SAFE. The earlier 394-line patch was duplicating that fix, so I removed it. The branch now adds one test file for #1023 and its joined control.

The test fails 4/5 on bb991c5 and passes 5/5 on current main, so it still detects the regression. The Markdown and parse-limit subset passes (1,289 tests), and ruff is clean.

The per-line tag-filter finding no longer applies: that filter is not part of this diff. I left the earlier review threads open because they refer to code that has been removed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

static_parse_limit when an inline code span wraps across a line break before a <placeholder>, turning clean Markdown into AE1 HIGH

3 participants